Methods of detection of software exploitation

ABSTRACT

A method for detecting software exploitation broadly comprises the steps of gathering information about processes and threads executing on a computing device, monitoring instructions executed by a thread that is currently running, performing the following steps if a function to create a process or a function to load a library is called, examining a thread information block, determining whether an address included in a stack pointer of the thread is in a range of addresses for a stack specified by the thread information block, and determining whether a first plurality of no-operation instructions is followed by shell code that is followed by a second plurality of no-operation instructions.

RELATED APPLICATION

The current patent application is a divisional patent application whichclaims priority benefit, with regard to all common subject matter, ofU.S. patent application Ser. No. 13/942,385, titled “METHODS OFDETECTION OF SOFTWARE EXPLOITATION”, filed Jul. 15, 2013. Theearlier-identified patent application is herein incorporated byreference in its entirety in the current patent application.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Embodiments of the current invention relate to the detection ofmalicious computer software.

2. Description of the Related Art

Software exploitation, also known as computer viruses, malware, spyware,adware, worms, and the like, is utilized by an attacker to gain accessto a user's computer system in order to obtain sensitive information,monitor the activities of the user, or control the operation of thesystem. The exploitation often occurs when the user receives data fromother parties or external systems such as while browsing the Internet orreceiving email. The exploitation may take advantage of security defectsin the programming of applications such as web browsers or emailreaders.

SUMMARY OF THE INVENTION

Embodiments of the current invention solve the above-mentioned problemsand provide a distinct advance in the art of the detection of maliciouscomputer software.

A first embodiment of the invention provides a method for detectingsoftware exploitation broadly comprising the steps of gatheringinformation about processes and threads executing on a computing device,monitoring instructions executed by a thread that is currently running,performing certain steps if a function to create a process or a functionto load a library is called. The steps performed may include examining athread information block, determining whether an address included in astack pointer of the thread is in a range of addresses for a stackspecified by the thread information block, and determining whether afirst plurality of no-operation instructions is followed by shell codethat is followed by a second plurality of no-operation instructions.

A second embodiment of the invention provides a method for detectingsoftware exploitation broadly comprising the steps of gatheringinformation about processes and threads executing on a computing device,monitoring instructions executed by a thread that is currently running,performing certain steps if a function to create a process or a functionto load a library is called. The steps performed may include examining aplurality of items on a stack, determining instructions that placeditems on the stack, determining whether the instructions include validsubroutine calls, and determining whether the instructions are locatedin an address space for executable code.

A third embodiment of the invention provides a method for detectingsoftware exploitation broadly comprising the steps of gatheringinformation about processes and threads executing on a computing device,monitoring instructions executed by a thread that is currently running,performing certain steps if a function to create a process or a functionto load a library is called. The steps performed may include examining aplurality of items on a stack, examining a chain of exception handlers,each exception handler including a first address pointing to the nextexception handler and a second address pointing to instructions forhandling an exception, and determining for each exception handlerwhether the second address is located in an address space for executablecode.

A fourth embodiment of the invention provides a method for detectingsoftware exploitation broadly comprising the steps of gatheringinformation about processes, threads, and applets executing on acomputing device, monitoring instructions executed by processes,threads, and applets that are currently running, monitoring any filethat is created by the applets, determining whether the file is beingexecuted as an additional process, and determining whether the file isbeing loaded as a library.

A fifth embodiment of the invention provides a method for detectingsoftware exploitation broadly comprising the steps of gatheringinformation about processes, threads, and applets executing on acomputing device, monitoring instructions executed by processes,threads, and applets that are currently running, utilizing a programminginterface, and determining whether a system.setsecuritymanager(null)call is made followed by a processbuilder.start( )call.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Other aspectsand advantages of the current invention will be apparent from thefollowing detailed description of the embodiments and the accompanyingdrawing figures.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

Embodiments of the current invention are described in detail below withreference to the attached drawing figures, wherein:

FIG. 1 is a view of a plurality of computing devices for detectingsoftware exploitation, as constructed in accordance with variousembodiments of the current invention;

FIG. 2 is a block schematic diagram of a processing element and a memoryelement, which are components of the computing devices of FIG. 1;

FIG. 3A is a flow diagram of a first portion of the steps of a methodfor detecting software exploitation in accordance with anotherembodiment of the current invention; and

FIG. 3B is a flow diagram of a second portion of the steps of the methodof FIG. 3A.

The drawing figures do not limit the current invention to the specificembodiments disclosed and described herein. The drawings are notnecessarily to scale, emphasis instead being placed upon clearlyillustrating the principles of the invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The following detailed description of the invention references theaccompanying drawings that illustrate specific embodiments in which theinvention can be practiced. The embodiments are intended to describeaspects of the invention in sufficient detail to enable those skilled inthe art to practice the invention. Other embodiments can be utilized andchanges can be made without departing from the scope of the presentinvention. The following detailed description is, therefore, not to betaken in a limiting sense. The scope of the present invention is definedonly by the appended claims, along with the full scope of equivalents towhich such claims are entitled.

In this description, references to “one embodiment”, “an embodiment”, or“embodiments” mean that the feature or features being referred to areincluded in at least one embodiment of the technology. Separatereferences to “one embodiment”, “an embodiment”, or “embodiments” inthis description do not necessarily refer to the same embodiment and arealso not mutually exclusive unless so stated and/or except as will bereadily apparent to those skilled in the art from the description. Forexample, a feature, structure, act, etc. described in one embodiment mayalso be included in other embodiments, but is not necessarily included.Thus, the current technology can include a variety of combinationsand/or integrations of the embodiments described herein.

Computing devices 10 for detecting software exploitation, constructed inaccordance with various embodiments of the current invention, are shownin FIG. 1. The computing devices 10 may include devices such as a servercomputer, a desktop computer, a work station computer, a laptopcomputer, and the like. Certain embodiments of the current invention maybe implemented in hardware, firmware, software, or combinations thereof.Each computing device 10 may comprise a processing element 12 coupledwith a memory element 14, as shown in FIG. 2.

The processing element 12 may include processors, microprocessors,microcontrollers, digital signal processors (DSPs), field-programmablegate arrays (FPGAs), analog and/or digital application-specificintegrated circuits (ASICs), and the like, or combinations thereof. Theprocessing element 12 may generally execute, process, or runinstructions, code, code segments, software, firmware, programs,applications, apps, processes, services, daemons, or the like, or maystep through states of a finite-state machine.

Typically, the processing element 12 comprises processors with an x86type architecture that includes components such as general purposeregisters, index registers, segment registers, base pointers, stackpointers, and the like. The processing element 12 may utilize aninstruction set that includes instructions from the x86 instruction set.Furthermore, the processing element 12 may execute a Windows& basedoperating system, produced by the Microsoft Corporation in Redmond,Wash., although other operating systems are also possible.

The memory element 14 may include data storage components such asread-only memory (ROM), programmable ROM, erasable programmable ROM,random-access memory (RAM), hard disks, floppy disks, optical disks,flash memory, thumb drives, universal serial bus (USB) drives, and thelike, or combinations thereof. The memory element 14 may include, or mayconstitute, a “computer-readable medium” or “computer-readable storagemedium” that is non-transitory in nature. The memory element 14 maystore the instructions, code, code segments, software, firmware,programs, applications, apps, services, daemons, or the like that areexecuted by the processing element 12. The memory element 14 may alsostore settings, data, documents, sound files, photographs, movies,images, databases, and the like. The processing element 12 may be incommunication with the memory element 14 through address busses, databusses, control lines, and the like.

The memory element 14 may include a physical address space and at leasta portion of a virtual address space and may be used to implement a filesystem. The memory element 14 may also be utilized to form one or morestacks, one or more heaps, and other data storage structures. A stackmay include a plurality of data storage units (typically memory addresslocations) that operate as a last-in, first-out (LIFO) component. Thestack may include a top address and a bottom address. The stacktypically stores data associated with function and subroutine calls. Astack pointer may be used in conjunction with the stack, such that thestack pointer usually contains the address of the next availablelocation in the stack. A heap may include a plurality of data storageunits that operate as a random-access storage area. The heap may beutilized for processing of data that is input from users or othersources.

At least a portion of the steps of a method 100 for detecting softwareexploitation, in accordance with an embodiment of the current invention,is shown in FIGS. 3A and 3B. The method 100 may be partially or whollyimplemented as a program that is generally stored in the memory element14 on a computer-readable medium and executed by the processing element12 in the computing device 10 when the computing device 10 is poweredon. Typically, a user is not aware that the method 100 is executing. Themethod 100 may be included in at least one process that is runningcontinuously on the computing device 10, although portions of the method100 may be executed only when certain events occur. For example, somesteps of the method 100 may be executed only when a running applicationattempts to execute another application, load a dynamically linkedlibrary (DLL), or perform a similar action that introduces new code tobe executed. The steps may be performed in the order as shown in FIGS.3A and 3B, or they may be performed in a different order. Furthermore,some steps may be performed concurrently as opposed to sequentially. Inaddition, some steps may be omitted.

With reference to step 101, information is gathered about currentlyrunning processes, threads, and loaded modules. A process may be aninstance of a program or application that is running. Threads andmodules may be portions of a program, and each program may include aplurality of both threads and modules. In various embodiments, themethod 100 gather information about all processes that are currentlyrunning. In other embodiments, the method 100 may gather informationabout processes that receive data from or communicate with sourcesexternal to the computing device 10, such as a local, metro, or widearea network, or the Internet. For example, the method 100 may gatherinformation about Internet access or browsing programs, electronic mailprograms, or the like. Each process, thread, and module may include anexecutable code section and a data section. The information gathered mayinclude information that is available in a thread information block(TIB), also known as a thread environment block (TEB), which is createdby the operating system. The TIB may include data such as a pointer to astructured exception handling frame, addresses of the top and the bottomof the stack, identification data (e.g., process ID, thread ID), and thelike. This information is specific to each thread and does not changeduring the execution of the thread.

The information gathered may also include a list of modules that arecomponents of each process along with an address range or position ofthe code section and the data section of each module within the virtualaddress space assigned to the process. The information on modules mayalso include the location of the module in the file system. Thisinformation may be utilized to exclude certain applications from beingsubjected to other steps of the method so as to avoid any potentialfalse positive exploitation detection results. The information gatheredmay further include a list of exported functions from each module andtheir addresses within the code section of the module.

With reference to step 102, the instructions that are executed by thecurrent process are monitored to determine whether functions to create aprocess or load a library are called. Examples of the functions include“CreateProcess” and “LoadLibrary” that are used in a Windows®-basedprogramming environment. If either function is called, then at least aportion of the remaining steps of the method is performed.

With reference to step 103, the stack pointer of the current thread isexamined to determine whether it is pointing outside of the rangespecified in the TIB. The range of virtual address locations for thestack of the current thread may be listed in the TIB. If the stackpointer contains an address that is not within the TIB listed range,then it may be noted in an internal log that the address in the stackpointer pointed outside of the range listed in the TIB. In someembodiments, a message may also be displayed to the user that a possibleexploit has been detected. In other embodiments, the call to create anew process or to load a library may be automatically blocked, or theprocess may also be terminated. Unless the process is terminated, atleast a portion of the remaining steps of the method 100 are performedto provide additional analysis. If the stack pointer includes a validaddress within the TIB listed range, then at least a portion of theremaining steps of the method 100 are performed.

With reference to step 104, the contents of the stack are examined andthe instructions that placed items on the stack are determined.Typically, when a process calls a subroutine, several items are placedon the stack between boundaries known as a frame. The frame is createdwhen the subroutine is called and destroyed when the subroutine iscomplete. At least one of the items in the frame is a return address towhich the flow of execution returns after the subroutine is complete.Usually, the return address is directly after the address of the commandthat called the subroutine.

With reference to step 105, the method 100 determines whether theinstructions include valid subroutine calls. The command or instructionat the address before the return address may be examined. Sincesubroutines may call other subroutines, the return address in eachframe, starting with the most recent one, is followed to examine theinstruction at the address before the return address. If any of theinstructions is not a valid subroutine call, then it may be noted in aninternal log that at least one instruction the addresses before returnaddresses is not a valid subroutine call. In some embodiments, a messagemay also be displayed to the user that a possible exploit has beendetected. In other embodiments, the call to create a new process or toload a library may be automatically blocked, or the process may also beterminated. Unless the process is terminated, at least a portion of theremaining steps of the method 100 are performed to provide additionalanalysis. If all of the instructions in the addresses before returnaddresses are valid subroutine calls, then at least a portion of theremaining steps of the method 100 are performed.

With reference to step 106, the method 100 determines whether theinstructions are located in an address space for executable code. Thereturn address in each frame is examined. If any return address is notin an executable code section of the process, then it may be noted in aninternal log that at least one return addresses is not located in anexecutable code section. In some embodiments, a message may also bedisplayed to the user that a possible exploit has been detected. Inother embodiments, the call to create a new process or to load a librarymay be automatically blocked, or the process may also be terminated.Unless the process is terminated, at least a portion of the remainingsteps of the method 100 are performed to provide additional analysis. Ifall of the instructions in the return addresses are located in validexecutable code sections, then at least a portion of the remaining stepsof the method 100 are performed.

With reference to step 107, the method 100 determines whether the stackincludes the return address of the DLL that created the thread.Typically, this may be NTDLL.dll, although other dynamic link librariesmay be used. If the return address is not found, then it may be noted inan internal log that the return address of the DLL that created thethread was not found. In some embodiments, a message may also bedisplayed to the user that a possible exploit has been detected. Inother embodiments, the call to create a new process or to load a librarymay be automatically blocked, or the process may also be terminated.Unless the process is terminated, at least a portion of the remainingsteps of the method 100 are performed to provide additional analysis. Ifthe return address of the DLL that created the thread is found, then atleast a portion of the remaining steps of the method 100 are performed.

With reference to step 108, the method 100 determines whether the bottomof the stack includes the address of the start of the thread. If thebottom of the stack does not include the address of the start of thethread, then it may be noted in an internal log that the bottom of thestack does not include the thread start address. In some embodiments, amessage may also be displayed to the user that a possible exploit hasbeen detected. In other embodiments, the call to create a new process orto load a library may be automatically blocked, or the process may alsobe terminated. Unless the process is terminated, at least a portion ofthe remaining steps of the method 100 are performed to provideadditional analysis. If the bottom of the stack does include the threadstart address, then at least a portion of the remaining steps of themethod 100 are performed.

With reference to step 109, a chain of exception handlers is examinedand it is determined whether any exception handler is outside the stack.An exception handler is a set of instructions for handling unusualsituations during the execution of a process, such as performing anindeterminate math function, receiving unexpected data from the user, orthe like. Typically, the exception handlers are stored on the stack in alinked list fashion. Thus, each exception handler may include a pointerto the set of executable instructions, in the executable code section,as well as a pointer to the next handler. The method 100 may determinewhether all pointers to the next handler point within the stack. If anypointers to the next handler point outside of the stack, then it may benoted in an internal log that a pointer to the next exception handler inthe chain of exception handlers points outside of the stack. In someembodiments, a message may also be displayed to the user that a possibleexploit has been detected. In other embodiments, the call to create anew process or to load a library may be automatically blocked, or theprocess may also be terminated. Unless the process is terminated, atleast a portion of the remaining steps of the method 100 are performedto provide additional analysis. If all of the next exception handlerpointers point inside the stack, then at least a portion of theremaining steps of the method 100 are performed.

With reference to step 110, the method 100 determines whether all of thepointers to the exception handler set of executable instructions arelocated in the executable code section. If any exception handler pointerdoes not point to the executable code section, then it may be noted inan internal log that at least one exception handler pointer does notpoint to the executable code section. In some embodiments, a message mayalso be displayed to the user that a possible exploit has been detected.In other embodiments, the call to create a new process or to load alibrary may be automatically blocked, or the process may also beterminated. Unless the process is terminated, at least a portion of theremaining steps of the method 100 are performed to provide additionalanalysis. If all of the exception handler pointers point to theexecutable code section, then at least a portion of the remaining stepsof the method 100 are performed.

With reference to step 111, the method 100 determines whether the chainof exception handlers forms a loop. In other words, it is determinedwhether any pointers to the next exception handler point to previousexception handlers. If so, then it may be noted in an internal log thata pointer to the next exception handler in the chain of exceptionhandlers points to a previous exception handler. In some embodiments, amessage may also be displayed to the user that a possible exploit hasbeen detected. In other embodiments, the call to create a new process orto load a library may be automatically blocked, or the process may alsobe terminated. Unless the process is terminated, at least a portion ofthe remaining steps of the method 100 are performed to provideadditional analysis. If none of the next exception handler pointerspoint to previous exception handlers, then at least a portion of theremaining steps of the method 100 are performed.

With reference to step 112, the memory addresses commonly used by heapsprays are examined and the method determines whether two nopsleds areseparated by shellcode. A heap spray is a technique that places acertain sequence of instructions at a predetermined location in theaddress space. Exemplary predetermined addresses may include 0x09090909and 0x0C0C000C. The instructions may include a plurality of no operation(NOP) instructions that form a “nopsled” or a “nopslide”. Heap sprayingmay further involve placing a first nopsled followed by shellcodefollowed by a second nopsled. The shellcode may include commands that anattacker wishes to execute. If two nopsleds are found that are separatedby shellcode, then it may be noted in an internal log that two nopsledshave been found that are separated by shellcode. In some embodiments, amessage may also be displayed to the user that a possible exploit hasbeen detected. In other embodiments, the call to create a new process orto load a library may be automatically blocked, or the process may alsobe terminated. Unless the process is terminated, at least a portion ofthe remaining steps of the method 100 are performed to provideadditional analysis. If the nopsled-shellcode-nopsled sequence is notfound, then at least a portion of the remaining steps of the method 100are performed.

With reference to step 113, any file that is created by an applet isexamined to determine whether it is being executed or loaded as alibrary. The applet may be a subprogram or subprocess that is typicallylaunched by an existing process as opposed to being directly executed bythe user. For example, the user may execute a web browser which in turnmay launch applets depending on the content of the web page beingviewed. An exemplary platform for launching applets is Java™ produced byOracle Corporation of Redwood Shores, Calif. The applet may create orreceive a file to be stored in the file system. If the file issubsequently executed as an additional process or loaded as a library,then it may be noted in an internal log that an applet-created file hasbeen executed or loaded as a library. In some embodiments, a message mayalso be displayed to the user that a possible exploit has been detected.In other embodiments, the call to create a new process or to load alibrary may be automatically blocked, or the process may also beterminated. Unless the process is terminated, at least a portion of theremaining steps of the method 100 are performed to provide additionalanalysis. If the applet-created file is not executed or loaded as alibrary, then at least a portion of the remaining steps of the method100 are performed.

With reference to step 114, a programming interface is utilized todetermine whether system.setsecuritymanager(null) is called followed byprocessbuilder.start( ). The programming interface may allow a processto inspect the state of and control the execution of applicationsrunning in a Java™ Virtual Machine, which is a run-time environment inwhich Java™ byte code can be executed. An exemplary programminginterface is the Java™ virtual machine tool interface (JVMTI). Thefunction system.setsecuritymanager(null) may remove or turn off Java™security measures for the process. The function processbuilder.start()may create a new process instance. If these two functions are called insuccession, then it may be noted in an internal log thatsystem.setsecuritymanager(null) was called followed byprocessbuilder.start( ). In some embodiments, a message may also bedisplayed to the user that a possible exploit has been detected. Inother embodiments, the call to create a new process or to load a librarymay be automatically blocked, or the process may also be terminated. Ifthe functions are not called, then no further actions may be taken.

Although the invention has been described with reference to theembodiments illustrated in the attached drawing figures, it is notedthat equivalents may be employed and substitutions made herein withoutdeparting from the scope of the invention as recited in the claims.

Having thus described various embodiments of the invention, what isclaimed as new and desired to be protected by Letters Patent includesthe following:
 1. A non-transitory computer-readable storage medium withan executable program stored thereon for detecting softwareexploitation, wherein the program instructs a processing element toperform the following steps: gathering information about processes andthreads executing on a computing device; monitoring instructionsexecuted by a thread that is currently running; and performing thefollowing steps when a function to create a process or a function toload a library is called examining a plurality of items on a stack,examining a chain of exception handlers, each exception handlerincluding a first address pointing to the next exception handler and asecond address pointing to instructions for handling an exception, anddetermining for each exception handler whether the second address islocated in an address space for executable code.
 2. Thecomputer-readable storage medium of claim 1, wherein the program furthercomprises the step of displaying a message to a user that a possiblesoftware exploit has been detected when any one of the second addressesis not located in the address space for executable code.
 3. Thecomputer-readable storage medium of claim 1, wherein the program furthercomprises the step of determining for each exception handler whether thefirst address is within the stack.
 4. The computer-readable storagemedium of claim 3, wherein the program further comprises the step ofdisplaying a message to a user that a possible software exploit has beendetected when any one of the first addresses is not located within thestack.
 5. The computer-readable storage medium of claim 1, wherein theprogram further comprises the step of determining for each exceptionhandler whether the first address points to a previous exceptionhandler.
 6. The computer-readable storage medium of claim 5, wherein theprogram further comprises the step of displaying a message to a userthat a possible software exploit has been detected when any one of thefirst addresses points to a previous exception handler.
 7. Anon-transitory computer-readable storage medium with an executableprogram stored thereon for detecting software exploitation, wherein theprogram instructs a processing element to perform the following steps:gathering information about processes and threads executing on acomputing device; monitoring instructions executed by a thread that iscurrently running; and performing the following steps when a function tocreate a process or a function to load a library is called examining aplurality of items on a stack, examining a chain of exception handlers,each exception handler including a first address pointing to the nextexception handler and a second address pointing to instructions forhandling an exception, determining for each exception handler whetherthe first address is within the stack, determining for each exceptionhandler whether the first address points to a previous exceptionhandler, and determining for each exception handler whether the secondaddress is located in an address space for executable code.
 8. Thecomputer-readable storage medium of claim 7, wherein the program furthercomprises the step of displaying a message to a user that a possiblesoftware exploit has been detected when any one of the first addressesis not located within the stack.
 9. The computer-readable storage mediumof claim 7, wherein the program further comprises the step of displayinga message to a user that a possible software exploit has been detectedwhen any one of the first addresses points to a previous exceptionhandler.
 10. The computer-readable storage medium of claim 7, whereinthe program further comprises the step of displaying a message to a userthat a possible software exploit has been detected when any one of thesecond addresses is not located in the address space for executablecode.
 11. A computing device for detecting software exploitation, thecomputing device comprising: a processing element coupled to a memoryelement, wherein the processing element is configured to detect softwareexploitation by: gathering information about processes and threadsexecuting on a computing device; monitoring instructions executed by athread that is currently running; and performing the following stepswhen a function to create a process or a function to load a library iscalled examining a plurality of items on a stack, examining a chain ofexception handlers, each exception handler including a first addresspointing to the next exception handler and a second address pointing toinstructions for handling an exception, and determining for eachexception handler whether the second address is located in an addressspace for executable code.
 12. The computing device of claim 11, whereinthe processing element is further configured to detect softwareexploitation by displaying a message to a user that a possible softwareexploit has been detected when any one of the second addresses is notlocated in the address space for executable code.
 13. The computingdevice of claim 11, wherein the processing element is further configuredto detect software exploitation by determining for each exceptionhandler whether the first address is within the stack.
 14. The computingdevice of claim 13, wherein the processing element is further configuredto detect software exploitation by displaying a message to a user that apossible software exploit has been detected when any one of the firstaddresses is not located within the stack.
 15. The computing device ofclaim 11, wherein the processing element is further configured to detectsoftware exploitation by determining for each exception handler whetherthe first address points to a previous exception handler.
 16. Thecomputing device of claim 15, wherein the processing element is furtherconfigured to detect software exploitation by displaying a message to auser that a possible software exploit has been detected when any one ofthe first addresses points to a previous exception handler.